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A METHOD FOR ENHANCING A DIGITAL IMAGE BASED UPON 

PIXEL COLOR 



FIELD OF THE INVENTION 

5 The invention relates generally to the field of digital image 

processing and, more particularly, to a method for enhancing the texture of a 
digital image. 

BACKGROUND OF THE INVENTION 

1 0 Traditional methods of increasing the apparent sharpness of a 

digital image, such as the technique of unsharp masking, often produce unwanted 
artifacts at large transition edges in the image. For example, unsharp masking is 
often described by the equation: 

Sproc = Sorg+B(Sorg-Sus) 

1 5 where Sproc represents the processed image signal in which the high frequency 
components have been amplified, Sorg represents the original image signal, Sus 
represents the imsharp image signal, typically a smoothed image signal obtained 
by filtering the original image, and B represents the high frequency emphasis 
coefficient. 

20 The unsharp masking operation may be modeled as a linear system. 

Thus, the magnitude of any frequency in Sproc is directly dependent upon the 
magnitude of that frequency in the Sorg image signal. As a consequence of this 
superposition principle, large edges in the Sorg image signal will often display a 
ringing artifact in the Sproc signal when the desired level of high frequency 

25 enhancement has been performed in other areas of the Sproc signal. This ringing 
artifact appears as a light or dark outline aroimd the large edge, and may be 
visually objectionable. 

Many non-linear filters based on local statistics exist for the 
purposes of noise reduction, sharpening, and contrast adjustment. For example, 

30 the median filter is well known in the art. In this filter, typically implemented for 
noise reduction, each pixel is replaced with the median value of some surrounding 
neighborhood. This filtering process is generally very successful at removing 
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impulse noise; however, the processed image appears slightly less sharp than the 
original. 

Another example of a non-linear filter based on local statistics is 
local histogram equalization, referred to as adaptive histogram modification by 
5 William Pratt on pages 278 - 284 of the book Digital Image Processing, Second 
Edition , John Wiley & Sons, 1991. With this filter, the values of pixels are 
modified by the cumulative histogram of a local window. This technique 
effectively adjusts the contrast of each region of a digital image, effectively 
increasing the local contrast in some regions of the image, and decreasing the 
['S 10 contrast in other regions. This technique does not intend to increase the apparent 

J[| sharpness of any given region. Also, this technique does not ensure that the typical 

artifacts of ringing will not occur. 

There exist many algorithms for sharpening the appearance of 
CS images without generating artifacts or enhancing the notability of noise. In U.S. 

Q 15 Patent 4,571 ,635, Mahmoodi and Nelson teach the method of deriving an emphasis 

;f coefficient B that is used to scale the high fi-equency information of the digital 

I y image depending upon the standard deviation of image pixel values in a local 

f5 neighborhood. In addition, in U.S. Patent 5,081,692, Kwon and Liang teach that 

the emphasis coefficient B is based upon a center weighted variance calculation. 
20 However, neither Mahmoodi et al nor Kwon et al consider the expected standard 
deviation of noise inherent in the imaging system. By not considering the noise 
inherent in the imaging system, both Mahmoodi and Kwon make the implicit 
assumption that all imaging sources and intensities have the same noise 
characteristics. In addition, neither use separate strategies for texture and edge 
25 regions. 

In U.S. Patent 4,794,531, Morishita et al teaches a method of 
generating the unsharp image with a filter whose weights on neighboring pixels are 
based upon the absolute difference between the central pixel and the neighboring 
pixel. Morishita claims that this method effectively reduces artifacts seen at the 
30 edges of a sharpened image (as compared with traditional unsharp masking). In 
addition, Morishita sets a gain parameter based upon local standard deviation and 
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the standard deviation of the entire image. Again, Morishita does not consider the 
levels of noise inherent in the imaging system in order to approximate signal to 
noise ratios. In addition, the method of Morishita does not offer explicit control 
over the amount of edge reshaping. 
5 In U.S. Patent 5,038,388, Song teaches a method of amplifying 

image details without amplifying the image noise by adaptively amplifying the 
high-frequency component of the image. An estimate of image noise power is 
used, however; this noise power is not described as being dependent on the 
intensity or the pixel. In addition. Song does not attempt to estimate signal to 

1 0 noise ratio in order to control the level of sharpening. 

In US. Patent 4,689,666 Hatanaka discloses a method of using the 
color characteristics of a color digital image for the purposes of reducing the noise 
component of the color digital image. Hatanaka describes a process of extracting 
color data for each picture element of the image, discriminating regions of the 

1 5 color digital image exhibiting a specific color on the basis of the extracted color 
data, and subjecting the image signal to spatial image processing for elimination 
of noise under different processing conditions for regions exhibiting the specific 
color and the remaining regions not exhibiting the specific color. Thus the 
method taught by Hatanaka has as a fundamental step the segmentation, or 

20 discrimination, of each pixel as belonging to the specific color or not belonging to 
the specific color. The step color discrimination can lead to unevenness the 
processed images due to the on/off nature of the color identification process. 

In US. Patent 5,682,443 Gouch and MacDonald disclose a method 
of processing color digital images for the purpose of spatial sharpness 

25 characteristic enhancement. A method of unsharp masking is described which 
separates each color channel of the original color digital image into two parts 
based solely on the spatial frequency content of the original color channel. The 
difference between the original color channel and a low spatial frequency 
component of the original color channel forms a fringe component, or high spatial 

30 frequency component of the original color channel. Gouch and MacDonald teach 
a method of modifying the fringe component based on the color of either the low 
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spatial frequency component or the original pixel values. The color considered is 
derived from the identical color channels that are sharpened. Their patent also 
discloses that the preferred method of implementing this feature uses a continuous 
mathematical function of color. The method disclosed by Gouch and MacDonald 
5 takes advantage of color as an image characteristic for enhancing the spatial 
detail. However, the unsharp masking procedure employed by Gouch and 
MacDonald has several shortcomings. First, their method only considers cases 
where each color channel of a color digital image undergoes the described unsharp 
masking operation. The variable weight used to modify the fringe data is derived 

1 0 from all of the color channels. Their method fails to consider cases when only a 
single digital image channel from a color digital image is sharpened, as is often 
the case when an image is represented as a luminance channel and a collection of 
chrominance channels. Additionally, Gouch and MacDonald necessitate that the 
modified fringe data is combined with the original pixel data. However, in many 

1 5 applications, it is preferred to modify the signal to which the modified fringe data 
is added. Thus, their method fails to consider cases where it is desirable to modify 
the signal to which the modified fringe data is combined. 

Thus, there exists a need for an alternative method of manipulating 
a digital image in order to generate an image signal that appears to be sharper, or 

20 more in focus, while minimizing the ringing artifact that is evident with the 

unsharp masking technique and enhancing the magnitude of detail in the scene in 
a noise sensitive manner. 

SUMMARY OF THE INVENTION 

25 It is an object of this invention to allow for independent control of 

the enhancement applied to detail, large edges, and noisy regions. 

The present invention is directed to overcoming one or more of the 
problems set forth above. Briefly summarized, according to one aspect of the 
present invention, the invention resides in a method for enhancing a color digital 

30 image by first identifying a local neighborhood of pixels including a pixel of 

interest and then using the color values of one or more of the pixels from the local 



neighborhood to calculate a color weighting factor for the pixel of interest, where 
the color weighting factor is a near continuous function of the location of the one 
or more local pixels from the local neighborhood in a color space. A spatial filter 
is applied to the values of pixels from the local neighborhood to calculate a first 
signal value and a second signal value for the pixel of interest. The color 
weighting factor is then used to modify either the first value or the second value, 
and an enhanced pixel value is then generated for the pixel of interest from a 
combination of the first value and second values. The process is then repeated for 
other pixels in the color digital image. 

The present invention has the advantage of boosting the texture 
signal by a factor that is related to an estimate of the local color in the image. 
Thus, the portion of the texture signal coincident with regions of the digital image 
channel having certain local colors, such as the green of a grassy field due to 
many blades of grass in addition to image system noise, will experience a greater 
level of boost as compared with the portion of the texture signal associated with 
certain other local colors, such as a large region of clear blue sky where the only 
modulation is likely to be noise resulting from the imaging system. Therefore, 
while it is not desirable to increase the amplitude of regions having only noise 
modulation, it is preferable to boost the modulation where it can be attributed to 
actual modulation in the scene. In the present invention, the local color functions 
as a classifier that may be used to distinguish between, e.g., the two 
aforementioned types of regions in a scene. 

These and other aspects, objects, features and advantages of the 
present invention will be more clearly understood and appreciated from a review 
of the following detailed description of the preferred embodiments and appended 
claims, and by reference to the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram illustrating an overview of the present 
method, including the invention as later described in more detail in connection 
with Figs. 11 and 15B. 
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Fig. 2 is a block diagram of the preference sharpener in Fig. 1 in 

more detail. 

Fig. 3 is a block diagram of a first embodiment of the sharpening 
processor shown in Fig. 2. 
5 Fig. 4 is a block diagram of a second embodiment of the sharpening 

processor shown in Fig. 2. 

Fig. 5 is a block diagram of a first embodiment of the pedestal 
splitter shown in Fig. 3. 

f Fig. 5 A is a block diagram of a second embodiment of the pedestal 
:i 1 0 splitter shown in Fig. 3. 

Cr§ Fig. 5B is a block diagram of a third embodiment of the pedestal 

splitter shown in Fig. 3. 

Fig. 6 is a block diagram of the avoidance signal generator shown in 

Fig. 5. 

1 5 Fig. 7 is an example of an artifact avoidance fimction applied by the 

avoidance signal generator shown in Fig. 6. 

Fig. 8 is a block diagram of the pedestal modifier shown in Fig. 3. 
Fig. 9 is an example of a portion of an image showing a central 
pixel and the associated local region of the image. 
20 Fig. 10 is an example of several tone scale fimctions. 

Fig. 1 1 is a block diagram according to the invention of the texture 
modifier shown in Fig. 3. 

Fig. 12 is a block diagram of a first embodiment of the local SNR 
estimator shown in Fig. 1 1 . 
25 Fig. 13 is a block diagram of a second embodiment of the local 

SNR estimator shown in Fig. 1 1 . 

Fig. 14 is a plot of a fimction used to generate a noise table. 
Fig. 1 5 A is a plot of a look up table (LUT) used by the SNR gain 
determiner shown in Fig. 1 1 . 
30 Fig. 1 5B is a plot of a look up table (LUT) used by the color gain 

determiner shown in Fig. 1 1 . 
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DETAILED DESCRIPTION OF THE INVENTION 

In the following description, the present invention will be described 
as a method implemented as a software program. Those skilled in the art will 
5 readily recognize that the equivalent of such software may also be constructed in 
hardware. Because image enhancement algorithms and methods are well known, 
the present description will be directed in particular to algorithm and method steps 
forming part of, or cooperating more directly with, the method in accordance with 
the present invention. Other parts of such algorithms and methods, and hardware 

1 0 and/or software for producing and otherwise processing the image signals, not 
specifically shown or described herein may be selected from such materials, 
components and elements known in the art. Given the system as described 
according to the invention in the following materials, software not specifically 
shown, suggested or described herein that is useful for implementation of the 

1 5 invention is conventional and within the ordinary skill in such arts. 

It is instructive to note that the present invention utilizes a digital 
color image which is typically a two-dimensional array of red, green, and blue 
pixel values corresponding to light intensities. As used herein the term digital 
image refers to the color image, and in particular to a whole two dimensional 

20 array, or any portion thereof that is to be processed. In addition, the preferred 
embodiment is described with reference to an image of 1024 rows of pixels and 
1536 lines of pixels, although those skilled in the art will recognize that digital 
images of different resolutions and dimensions may be used with equal, or at least 
acceptable, success. With regard to matters of nomenclature, the value of a pixel of 

25 a digital image located at coordinates {x,y), referring to thejc''' row and thejv'^ 

column of the digital image, shall herein comprise a triad of values [r(x,y), g(x,y), 
b(x,y)] respectively referring to the values of the red, green and blue digital image 
channels at location (x,y). In this regard, a digital image may be considered as 
comprising a certain number of digital image channels. In the case of a digital 

30 image comprising red, green and blue two-dimensional arrays, the image 

comprises three channels, namely, red, green and blue channels. Additionally, a 



luminance channel n may be fomied from the color channels. The value of a pixel 
of a digital image channel n located at coordinates {x,y), referring to the x^^ row and 
they^ column of the digital image channel, shall herein be a single value referred 
to as n(x,y), 

5 Referring to the overview of the invention shown in Fig. 1 , a digital 

image is input to a preference sharpener 2, which sharpens the input digital image 
for the preference of an operator. The purpose of the preference sharpener 2 is to 
enhance the detail present in the digital image without enhancing noise or creating 
artifacts. This purpose is accomplished by a process that decomposes a digital 

1 0 image channel into a signal corresponding to the image detail, a signal 

corresponding to the image edges, a signal corresponding to local color and a 
signal corresponding to the image signal to noise ratio (SNR), as will be described 
in detail hereinbelow. This decomposition and creation of separate signals allows 
for independent control of the enhancement applied to detail, large edges, and 

1 5 noisy regions. The output of the preference sharpener 2 comprises a digital image 
that, in accordance with the invention, appears sharper and more natural than can 
be achieved with traditional means of sharpening a digital image. The particular 
approach of this invention, which relates to the independent control of the 
enhancement applied to detail based on local color, will be described in detail in 

20 relation to Figxires 1 1 and 15B after other elements of the preferred embodiment 
have been described in relation to the intervening Figvues. 

Fig. 2 shows a block diagram of the preference sharpener 2. The 
digital color image and noise information (in the form of a noise table 6) are input 
to the preference sharpener 2. The noise information is input in the form of a 

25 luminance table. The digital image is input to a luminance/chrominance converter 
10 in order to generate a luminance digital image channel n(x,y) and two 
chrominance digital image channels gm(x,y) and ill(x,y). The luminance digital 
image channel n(x,y) and the luminance noise table are input over respective lines 
11a and 1 lb to a sharpening processor 20 for enhancing the digital image channel 

30 n(x,y) based on knowledge of the system noise. The transformation performed in 
the converter 10 from the digital color image, typically in an RGB space 
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comprising separate red, green and blue image channels, to a luminance- 
chrominance color space is done by means of a color space matrix transformation, 
which results in the luminance digital image channel n(x,y) and the two 
chrominance digital image channels gm(x,y) and ill(x,y),as is well known in the art. 
5 In accordance with the invention, as will be explained later, the two chrominance 
digital image channels gm(x,y) and ill(x,y ) are also input to the sharpening 
processor 20. In the preferred embodiment, the matrix transformation to the 
luminance-chrominance space may be described with the following equation: 
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S^J Equation Al provides a matrix rotation from RGB space into a luminance- 

5 chrominance space in order to isolate a single channel upon which the tone scale 

function (to be described) can operate. For example, a film scanner records (for 
M each pixel) the red, green and blue printing density at that point. These three 

n 15 values can be thought of as a pixel's location in three-dimensional space. The 

converter 10 performs an axis rotation as shovra in equation Al, which provides 
for a neutral axis, upon which R = G = B, and two color difference axes, green- 
magenta and illuminant. Note that rather than using the matrix rotation specified 
with Equaiton Al, standard color space descriptions such as CIE L*A*B* or CIE 
20 L*C*H* may alternatively be determined by the Ixmiinance/chrominance converter 
10. These color spaces are well known in the art of image processing. 

The noise table input to the sharpening processor 20 provides the 
relationship between the signal intensity level / and the expected amoimt of noise 
aN(0 for that intensity level. In the preferred embodiment, as will be described in 
25 more detail, the noise table is a two colimin table, wherein the first column 
represents an intensity level / and the second column represents the expected 
standard deviation of noise an(0 for that intensity level. 

The chrominance digital image channels gm(x,y) and ill(x,y) are 
input over a line 1 Ic to a chrominance processor 40 and may be adjusted as 
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desired. For instance, the action of the chrominance processor 40 may be to scale 
the chrominance image channels by a constant greater than 1.0 in order to increase 
their apparent saturation. The operation of the chrominance processor is not of 
particular relevance to the present invention, and consequently it will not be 
5 discussed further, except to note that, in the present embodiment, the output of the 
chrominance processor 40 is maintained identical to its input. 

The digital image channels output from the sharpening processor 20 
and the digital image channels output from the chrominance processor 40 are input 
to an RGB converter 30 for conversion back to a digital image composed of a red, 

1 0 green, and blue digital image channels. This conversion is again accomplished 
with a matrix rotation (i.e., the inverse of the previous color rotation matrix 
performed by the converter 10). Inverting a 3 by 3 matrix is well known in the art 
and will not be further discussed. The output of the RGB converter 30 is a digital 
image that has been sharpened for the preference of the operator. 

15 As described in the preferred embodiment, the sharpening processor 

20 operates upon only a luminance digital image channel. However, as an 
alternative embodiment, the sharpening processor could be applied to each of the 
red, green, and blue digital image channels. In this case, the digital color image 
signals (which are RGB signals) would be directly applied to the sharpening 

20 processor 20. 

In Fig. 3, which illustrates a block diagram of the sharpening 
processor 20, the digital image channel n(x,y) is shown to be separated into two 
portions by a pedestal splitter 50, that is, into a pedestal signal and a texture signal. 
The preferred embodiment of the pedestal splitter 50, shown in Fig. 5, outputs a 

25 texture signal ntxt(x,y) and a pedestal signal nped(x,y). The texture signal ntxt(x,y) 

primarily contains the detail of the image, as well as the image noise. The pedestal 
signal nped(x,y) is a conceptually smooth signal, except at regions of the image 
containing large occlusion boundaries, where the pedestal signal contains the 
transition edge. In the preferred embodiment, the sum of the textiu*e and the 

30 pedestal signals is the digital image channel. For this reason, the digital image 

channel is said to have been separated into two channels which together essentially 
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comprise the digital image channel. An ahemative embodiment of the pedestal 
splitter 50, shown in Figure 5A, essentially comprises a linear FIR filter. In this 
alternative embodiment, the pedestal signal nped(x,y) is essentially equivalent to a 
lowpass signal and the texture signal ntxt(x,y) is essentially equivalent to a highpass 
signal. Since other spatial fi*equency divisions are possible, those skilled in the art 
v^ill understand that the present invention will yield beneficial results for any 
number of definitions of texture signals and pedestal signals. For example. Figure 
5B describes an alternative pedestal splitter which utilizes a non-linear filter in 
order to separate a digital image channel into a pedestal signal nped(x,y) and a 
texture signal ntxt(x,y). 

Referring again to Fig. 3, the digital image channel n(x,y), the 
texture signal ntxt(x,y) and the noise table 6 are input to a texture modifier 70, The 
purpose of the texture modifier 70 is to enhance the magnitude of detail in the 
scene in a noise sensitive manner. An estimate of the local signal/noise ratio 
(SNR) is made using information about the expected standard deviation of noise 
for a given intensity level as supplied by the noise table. This estimate of the local 
SNR is used to set the boost factor relating to the level of enhancement given to a 
local level of the texture signal. This procedure will be described in detail 
hereinbelow. The output of the texture modifier 70 is an enhanced texture signal 
i^*txt(x,y). Furthermore, the pedestal signal nped(x,y) is input to a pedestal modifier 
60 for the purpose of increasing edge contrast to make the edges appear to have 
greater clarity and sharpness. The purpose of the pedestal modifier 60 is to 
enhance the image edges without producing artifacts. The method employed by 
the pedestal modifier 60 will be described in detail hereinbelow. The output of the 
pedestal modifier is an enhanced pedestal signal n'ped(x,y)^ 

The outputs of the pedestal modifier 60 and the texture modifier 70 
are input to an adder 80, which adds the enhanced texture signal n*txt(x,y) and the 
enhanced pedestal signal n ped(x,y) together in order to produce the digital image 
channel output fi*om the sharpening processor 20. The enhanced digital image 
channel n *(x,y) may be expressed as 
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n *(x,y) = n 'txt(x.y) + n ped(x,y) 

Fig. 4 shows an alternative embodiment of the sharpening processor 
20, illustrating that minor perturbations to the arrangement of the components 
5 often have only a slight effect on the output of the sharpening processor 20. In this 
regard, the pedestal splitter 50 outputs the texture signal ntxt(x.y) and an avoidance 
signal a(x,y). The avoidance signal a(x,y) is an intermediate signal calculated by 
the pedestal splitter 50 for determining the location of edges in the image. The 
derivation of this signal will be described below. The value of the avoidance 

1 0 signal ranges from 0.0 to 1.0. hnage locations (x,y) having a value of the 
avoidance signal a(x,y) = 0.0 correspond with edge regions of the image; 
conversely, image locations (x,y) having a value of the avoidance signal a(x,y) > 
0.0 correspond with regions of image detail or noise. As mentioned before, it is an 
object of this invention to allow for independent control of the enhancement 

1 5 applied to detail, large edges, and noisy regions. Thus, in the digital image 
channel output from the sharpening processor 20, it is desired that the edges 
(where a(x,y) = 0) be enhanced by the pedestal modifier 60, and that the regions of 
image detail or noise (corresponding to those locations where a(x,y) > 0.0) be 
enhanced by the texture modifier 70. 

20 To this end, the digital image channel n(x,y) is input to the pedestal 

modifier 60 and as before the textiu-e signal ntxt(x,y) is input to the texture modifier 
70. The two resulting outputs from the pedestal modifier 60 and the texture 
modifier 70, the enhanced texture signal n*txt(x,y) and the enhanced pedestal signal 
nped(x,y)^ are input to an avoidance adder 81. The avoidance adder 81 requires 

25 three inputs: the two signals n'txt(x,y) and n*ped(x,y) to be summed, and the artifact 
avoidance signal a(x,y). The two signals to be summed undergo a transformation 
such that one of the signals to be summed is multiplied by a(x,y), and the other is 
multiplied by (l-afx,j;^). The two transformed signals are then added. The signal 
input which is scaled by a(x,y) is known as the ''a(x,y) input" of the avoidance 

30 adder 81 and the signal input which is scaled by {\'a(x,y)) is known as the "(1- 

a(x,y)y' input of the avoidance adder 81 . In the case of the present embodiment of 
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the sharpening processor 20, the signal output from the avoidance adder can be 
expressed as: 

^Yx^y) = a(x,y)ntjct(x,y) + {\-a(x,y))nped(x,y) 

5 

Referring to Fig. 5, the digital image channel n(x,y) input into the 
pedestal splitter 50 is split into a high pass signal nhp(x,y) and a low pass signal 
nip(x,y) by a frequency splitter 94. Although there are many known techniques for 
generating highpaiss and lowpass signals, the frequency splitter is preferably 

1 0 implemented with a Gaussian filter having a standard deviation (sigma) of 0.9 
pixels. The preferred value of the standard deviation of the Gaussian filter may 
vary with image size. The value of 0.9 pixels for the standard deviation was 
derived by optimizing the present invention with 1024 by 1536 pixel size images. 
This Gaussian filter is a two dimensional circularly symmetric low-pass filter 

1 5 whose filter coefficients may be derived by the following formula which is well 
known in the art: 

g(ij) = l/(sigma sqrt(27r)) exp[-(i2+j2)/(2 sigma^)] 
Where g(ij) = the Gaussian filter coefficient at the (hj)^ pixel 
20 sigma = the standard deviation of the Gaussian filter (0.9) 

71 = the constant approximately 3.1415. . . 

In the preferred embodiment, / and j range from -3 to +3, inclusive, for a total of 
49 coefficients for the filter g(ij). Note that another technique, well known in the 
25 art, exists for separating a Gaussian filter into horizontal and vertical components 
for subsequent application for the purpose of computational cost reduction. In 
either case, the frequency splitter 94 applies the Gaussian filter g(ij) to the digital 
image channel n(xy) by the process of convolution, which is well known in the art. 
The convolution may be expressed as: 

30 

nip(x,y) = ZZ n(X'i,y-j)g(ij) 
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with the summations occurring over all i and j. The signal nip(x,y) resulting from 
this convolution process is the low-pass signal. The low-pass signal is output from 
the frequency splitter 94. In addition, the high-pass signal nhp(x,y) is output from 
5 the frequency splitter 94 after having been derived by the following relationship: 

^hp(x.y) = ri(x,y) - nip(x,y) 

The low-pass signal nip(x,y) is input into an avoidance signal 
1 0 generator 104 for forming the avoidance signal a(x,y), as will be described in detail 
below. A texture signal generator 90 receives both the high-pass signal nhp(x,y) 
and the avoidance signal a(x,y) and both signals are multiplied therein for 
producing the texture signal ntxt(x,y). Thus, the texture signal ntxt(Xyy) may be 
expressed as: 

15 

^txt (x,y) = a(x,y) * nhp(x,y). 

The texture signal ntxt(x,y) computed by the texture signal generator 90 is then 

output by the pedestal splitter 50. In addition, as shown in broken line in Fig. 5, 
20 the avoidance signal a(x,y) may optionally be output by the pedestal splitter 50, in 

particular to provide an input to the avoidance adder 81 in the alternative 

embodiment of the sharpening processor 20 shown in Fig. 4. 

A pedestal generator 100 receives the luminance digital image 

channel n(x,y) and the texture signal ritxt (x,y) and subtracts the texture signal from 
25 the luminance signal, which produces the pedestal signal nped(x,y). Thus, the 

pedestal signal may be expressed as: 

rtped (x.y) = n(x,y) - ntxt (x,y)> 

30 The pedestal signal nped(x,y) computed by the pedestal signal generator 100 is then 
output from the pedestal splitter 50. 
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An alternative embodiment of the pedestal splitter 50 (identified as 
50A) is shown in Fig. 5A, The luminance digital image channel n(x,y) input into 
the alternative embodiment of the pedestal splitter 50A is split into a high pass 
signal and a low pass signal by a frequency splitter 94 preferably with the 
5 Gaussian filter as described above having a standard deviation (sigma) of 0.9 
pixels. The preferred value of the standard deviation of the Gaussian filter may 
vary with image size. The value of 0.9 pixels for the standard deviation was 
derived by optimizing the present invention with 1024 by 1536 pixel size images. 
The frequency splitter 94 applies the Gaussian filter g(i J) to the digital image 
1 0 channel n(x,y) by the aforementioned process of convolution. 

The signal nip(x,y) resulting from this convolution process is the 
low-pass signal output from the frequency splitter 94. In addition, a high-pass 
signal nhp(x,y) is output from the frequency splitter 94 after having been derived by 
the following relationship: 



15 



nhp(x.y) = n(x,y) - nip(x,y) 



In this alternative embodiment of the pedestal sphtter, the lowpass 
signal nip(x,y) is output from the pedestal splitter as the pedestal signal nped(x,y). 
20 (i.e., in this case, the pedestal signal nped(x,y) is set equal to the lowpass signal 

nip(x,y) rather than being derived with a control signal. This is equivalent to a(x,y) 
= 1.) Likewise, the texture signal ntxt(x,y) is set equal to the highpass signal 
nhp(x.y)' 

This altemative embodiment of the pedestal splitter 5 OA simply 
25 performs a digital filtering operation by convolution and outputs the highpass and 
lowpass signals as the texture and pedestal signals, respectively. This embodiment 
of the pedestal sphtter 50A is simpler than the preferred embodiment 50 because 
no control signal is used or calculated. However, the quality of the digital image 
output from the preference sharpener 4 employing this altemative embodiment is 
30 inferior to the quality of the digital image output from the preference sharpener 4 
employing the preferred embodiment of the pedestal splitter 50. 
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Those skilled in the art will recognize that there are many types of 
operations that may be used to generate two signals that essentially sum to 
generate the digital image channel. The preferred embodiment of the pedestal 
splitter and the alternative embodiment of the pedestal splitter are both examples 
5 of operations that may be used to separate a digital image channel into two signals 
that essentially simi to generate the digital image channel. 

Referring to Fig. 6, there is illustrated a block diagram of the 
avoidance signal generator 104. In this regard, the low pass signal nip(x,y) is input 
into a non-directional squared gradient calculator 106 for producing a non- 

1 0 directional gradient signal. This calculation is performed by first calculating the 
difference between the pixel and its upper vertical neighbor, as well as the 
difference between the pixel and its horizontal neighbor to the right. The non- 
directional squared gradient is the sum of the squares of these two differences. 
The non-directional squared gradient signal, ndg(x,y) may be expressed by the 

1 5 following equation: 

ndg(x,y) = \nip(x,y)-nip(x-Ly) f + [nip(x,y)'nip(x,y-^l) f 

The value of ndg(x,y) is then mapped by an artifact avoidance function applicator 
20 108 to produce an artifact avoidance signal a(x,y) as the output of the avoidance 
signal generator 104. 

Referring to Fig. 6, this mapping is performed by passing the non- 
directional squared gradient signal, ndg(x,y), through an artifact avoidance 
function av(y) formed in the following manner. The preferred embodiment 
25 utilizes an artifact avoidance function defined by: 

av(y) = (1/2) (1 + COS(7r(y-Co)/(C^-Co)) 

for y > Co and y < Cj, 

30 

av(y) = 0 
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for y >= and 

av(y) = 1 

5 

for y <= Co where Co and Cj are numerical constants. 

The preferred values for Co and C^ will vary according to the range 
of the input data. The input data range in the images used in the development of 
this preferred embodiment ranged from 0 to 4095. In this case, the preferred value 
1 0 for Co is 996, and the preferred value for Cj is 8400. As an altemate method for 
constructing the artifact avoidance function is described in the following equation: 

av(y) = l-sqrt((y-Co)/(C,-Co)) 

15 for y >= Co and y <= Cj, 

av(y) = 1 

for y < Co, and 

20 

av(y) = 0 

for y > Cj. Fig. 7 shows an example plot of the ftmction av(y). 

Note that in the preferred embodiment, the value of C^ is related to 
25 the size sigma (in standard deviations) of the Gaussian filter. The value of Cj is 
inversely proportional to the square of sigma (in standard deviations) of the 
Gaussian filter. Preferably, C^ may be determined by the relation: 

Cj = 6804/(sigma*sigma) 

30 

Also, Co is related to Cj by the equation: 
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Co = 0.127 Cj- 18 

The artifact avoidance signal, a(x,y), is generated by the artifact 
5 avoidance function appHcator 108. This is accompUshed by applying the artifact 
avoidance function av(y), to the non-directional gradient signal ndg(x,y), as shown 
in Figure 7. The mathematical result is described by the equation: 

a(x,y) = av ( ndg(x,y) ). 

10 

The artifact avoidance function shown in Figure 7 is most efficiently implemented 
as a look-up-table (LUT). 

It is helpful to understand that the avoidance signal a(x,y) is an 
example of a control signal created from a spatially filtered version of the digital 
1 5 image channel. To begin with, a generalized linear spatial filter is described by the 
equation 

c(x,y) = ZIJd(x-Uy-j)g(ij) 

where the d(x-Uy'j) values represent local pixel values surrounding the (x,y)^^ pixel 
20 and the g(ij) values represent numerical coefficients of a digital filter that do not 
depend on the pixel values, and are independent of x and j^. A non-linear spatial 
filter herein described shall be defined as a spatial filter which cannot be described 
by the linear spatial filter equation. The output of applying a generalized control 
signal to an input signal bears a multiplicative relationship to the input signal. An 
25 example of applying a control signal is given by the equation: 

c(x,y) = h(x,y)d(x,y) 



30 



where the d(x,y) values represent the input signal (x,y)^^ pixel values and the 
h(x,y) values represent the (x,y)^^ pixel values of the control signal. The result of 
applying a control signal to an input signal falls into the general category of a non- 
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linear spatial filter if the control signal is derived from a spatial filtered version of 
the input signal. The avoidance signal a(x,y) is an example of a control signal 
created fi-om a spatially filtered version of the digital image channel. The texture 
signal ntxt(x,y) described in the preferred embodiment is an example of a non- 
5 linear spatial filter produced with the application of a control signal and a digital 
filter applied to the image channel n(x,y). 

Additionally, several other examples of non-linear filters which are 
well known in the art of digital image processing occur when the non-linear filter 
coefficients g(ij) are not independent of position x and y. More specifically, it is 
%j 1 0 often the case that the non-linear filter coefficients g(ij) are dependent upon the 

J rl value of pixels of d(x,y) within a local neighborhood of (x,y). In the preferred 

l^, embodiment, the sum of the coefficients of g(i J) is 1 .0, although this should not 

I LI 

r n be seen as a limitation to the scope of the embodiment. Note that the median filter 

which is well known in the art of image processing is an example of a non-linear 

? 3 15 filter. In addition, the sigma filter described by Jong-Sen Lee in the article 

"Digital Image Enhancement and Noise Filtering by use of Local Statistics," IEEE 
Transactions on Pattern Analysis and Machine Intelligence^ Vol. PAMI-2, No. 2, 

C3 March 1980, is also an example of a non-linear filter. Note that in this description 

the non-linear filter is described as being only a fixnction of the image signal 
20 d(x,y), and thus is a Finite Impulse Response (FIR) filter. The non-linear filter 

may similarly be a recursive filter for which the coefficients of the recursive filter 
are dependent upon location (x,y). For example, in commonly-assigned, 
copending Patent Application Serial No. 09/457,036, filed December 8, 1999 and 
entitled "Adjusting the Contrast of a Digital Image with an Adaptive Recursive 
25 Filter", therein is described an adaptive recursive filter for which the filter 

coefficients vary depending upon the local pixel values in a neighborhood of (x,y). 
Fig 5B describes an altemative pedestal splitter which utilizes a non-linear filter in 
order to separate a digital image channel into a pedestal signal nped(x,y) and a 
texture signal ntxt(x,y). In this altemative pedestal splitter, the digital image 
30 channel n(x,y) is input to the pedestal generator 5 1 . The pedestal generator 5 1 
calculates a pedestal signal nped(x,y) from the digital image channel n(x,y) by 



application of a non-linear filter to the digital image channel. The texture signal 
^txi(x,y) is then calculated by passing the negative of the pedestal signal nped(x,y) 
and the digital image channel to an adder 53, as shown. The adder 53 determines 
the texture signal titxt (x,y) by applying the following formula: 
5 ritjct (x,y) = n(x,y) - riped (x,y). 

In this manner, the pedestal splitter 50B generates a pedestal signal nped(x,y) and a 
texture signal w^i (x,y)^ The pedestal signal nped(x,y) and the texture signal ntxt(x,y) 
are output fi*om the pedestal splitter 5 OB. 

Referring now to Fig. 8, the pedestal modifier 60 shown therein is a 

1 0 sharpening filter algorithm that is adaptive to the morphology of a digital image 
channel, as based on one or more image characteristics calculated over a localized 
region of the digital image channel (as isolated by a two-dimensional window). 
The filtering algorithm includes a stage for normalizing the tone scale conversion 
with a scaling fimction for the image statistic(s) over the localized region of a 

1 5 digital image channel and then using the scaled tone scale to reshape the structure 
of edges in the digital image channel. More specifically, the shape of the local 
tone scale fiinction is ordinarily selected before processing the image, but the 
exact scaling and translation of the local tone scale operation is determined fi-om 
the statistics in any localized region. Accordingly, after the region is isolated, 

20 implementation of the algorithm includes identifying a desired statistical 

characteristic from the local region, performing a local tone scale conversion that 
is normalized in relation to the statistical characteristic, and mapping the central 
pixel of the local region through the tone scale conversion to generate an 
enhanced central pixel value. More detail of the pedestal modifier 60 is described 

25 in commonly assigned U.S. Patent Application Serial No. 09/324,239, enfitled "A 
Method for Enhancing the Edge Contrast of a Digital Image" and filed June 2, 
1999 in the names of A.G. Gallagher and E.B. Gindele, and which is incorporated 
herein by reference. 

The pedestal modifier 60 may be implemented in several modes. 

30 More specifically, and particularly as shown in the block diagram of Figure 8, the 
pedestal modifier 60 is preferably practiced in a first mode by scaling the input 
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pixel values to the local maximum and minimum for each region and then applying 
the scaled input values to a tone scale function with normalized coordinates; the 
output obtained by this conversion operation is then processed with an inverse of 
the scaling function (used to generate the input) to obtain the enhanced output 
5 value. Alternatively, in a second mode of implementing the pedestal modifier 60, 
a local tone scale is constructed having coordinates that are boxmded by the local 
maximum and minimum for each region, and then each central pixel value is 
mapped through the bounded coordinates of the tone scale function. This will 
directly produce the enhanced output value. In a third mode, only the input of the 

1 0 tone scale function is scaled with the inverse scaling function and the output pixel 
is processed with the inverse of the scaling function; in a fourth mode, only the 
output of the tone scale function is scaled and the input pixel is processed with the 
scaling function. In any mode, the shape of the tone scale function may remain 
fixed over the image; however, it may be desirable to vary the shape of the tone 

1 5 scale function dependent upon the statistical characteristics of the region. 

Furthermore, while it may be typical to apply the normalized tone scale operation 
to substantially all the pixels in the input digital image channel, it may be desirable 
to only apply the method to selected areas of the digital image channel. For details 
of the second, third and fourth modes, the reader is directed to the aforementioned 

20 Serial No. 09/324,239, which is incorporated herein by reference. These 

alternative modes provide a processed digital image channel identical to the 
processed digital image channel obtained by the pedestal modifier 60 disclosed in 
the preferred embodiment (i.e., the first mode. Fig. 8). 

Referring further to Fig. 8, there is illustrated a block diagram of a 

25 preferred embodiment for implementing the pedestal modifier 60 according to the 
first mode as set forth above. It also facilitates understanding to note that the 
preferred embodiment of the pedestal modifier 60 is described in reference to a 
relatively high resolution digital image channel having, e.g., dimensions of 1024 
pixels in height by 1536 pixels in width, although the invention is similarly 

30 operative with image channels having greater or lesser image resolutions. The 
choice of image resolution, however, may have a concomitant effect on the 
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window size, i.e., while a 5 X 5 window has been found preferable for 1024 X 
1536 resolution, the choice of a higher resolution image sensor would ordinarily 
allow a larger window size, and vice versa for a lower resolution sensor. 

With reference also to Fig. 9, the value of each input central pixel 
5 1 10 of the digital representation of the digital image channel is first passed to a 
local statistics determiner 116 for calculating local statistics. The values of 
surrounding pixels 114 included within a local region 112 are also passed to the 
local statistics determiner 116. The local region 112 and the input pixel 110 
constitute a spatial region of the digital image channel. In the preferred 

1 0 embodiment, the local region about each input pixel 1 10 is defined as a square 
neighborhood centered on the input pixel for which the local statistics are being 
determined. In the preferred embodiment, the local region is a square 
neighborhood with 5 pixels on each side. Figure 9 illustrates the local region 
implemented in the preferred embodiment. The pixels 114 included within the 

1 5 local region 112 are illustrated as the 24 pixels (indicated by hatching) surrounding 
the input pixel 110 passed to the local statistics determiner 116. Those skilled in 
the art will recognize that the present invention will be operable with a large 
variety of local region sizes and shapes, and the usage (in this description and as 
claimed) of the term "central" to describe the processed pixel merely refers to a 

20 pixel having a local surroimd of pixels, rather than a pixel at the centroid of its 
local region. 

In Figure 8, the value u of each central pixel 1 10 is input to the 
local statistics determiner 116 and a scaling function evaluator 120. In addition, 
the values of the pixels 1 14 in the local region are input to the local statistics 

25 determiner 1 16 in order to generate a statistical characteristic of the local region. 
A scaling function generator 118 uses the statistical characteristic to generate a 
scaling function s(x) that is applied to the scaling function evaluator 120 in order 
to normalize the value u of each input pixel 110 and thereby generate a first 
intermediate value I. The statistical characteristic is also used by an inverse 

30 scaling fimction generator 122 to generate a function s'^x) that is the inverse of the 
scaling function s(x). The first intermediate value I is then applied to a tone scale 
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applicator 124, which maps the intermediate value I through a tone scale function 
f(x) provided by a tone scale function generator 126 in order to generate second 
intermediate value h. The output intermediate value I2 fi*om the tone scale 
applicator 124 is then applied to an inverse scaling function applicator 128, which 
5 generates a processed value u* in the system image space by using the inverse 
function s"\x) provided by the inverse scaling function generator 122. 

More specifically, the local statistics determiner 116 calculates 
several characteristics that describe the image data contained in the local region 
112 and the input pixel 110. The local statistics could be any mathematical 

1 0 combination of the pixel values contained in the local region 112 and the input 
pixel 110. In the preferred embodiment, the local statistics determiner 116 
determines two local statistics, the maximum and minimum pixel values from the 
local region 112 and the input pixel 110. As an altemative, the local statistics 
determiner 116 could determine the mean of the n maximum pixel values, and the 

1 5 mean of the m minimum pixel values, where n and m are integers much smaller 
than the number of pixels contained in the local region 112, for example, n = m — 
3. As will be described, these local statistics are utilized in the generation of the 
scaling function s(x) and the inverse scaling function s"*(x). 

The local statistics determined by the local statistics determiner 116 

20 are then passed to the scaling function generator 118 and the inverse scaling 

function generator 122, In the preferred embodiment, the scaling function may be 
represented with the following equation: 

s(u)= 

X-N 

25 where X represents the maximum pixel value from the local region 112 and N 

represents the minimum pixel value from the local region 112. The purpose of the 
scaling function is to modify the value of the central pixel 110 such that the 
modified pixel value may be used as an input to a tone scale function 130, as 
shown in Figm-e 10. In the preferred embodiment, this tone scale function 130 is 

30 applied as a look-up-table (LUT), In the preferred embodiment, the effect of the 
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scaling function is to normalize (i.e., compress) the interval [N X] of input values 
to an interval [0 1], as shown in Fig. 10. 

In a similar manner, the inverse scaling function generator 122 
creates the inverse scaling function s"Vu); the inverse scaling function s"*(u) may 
5 be represented by the following equation: 

s-'(u)=u(X-N)-hN 

The purpose of the inverse scaling function is to scale the output values obtained 
1 0 from the tone scale function 130 back to the interval [N X]. The scaling function 
and the inverse scaling function are indeed inverses, as it can easily be shovm that 
s-'(s(u)) = u. 

The value of the input pixel 1 10 is then passed to the scaling 
function evaluator 120 in order to generate an intermediate value I = s(u), where I 

15 is the intermediate value, s(x) is the scaling function, and u is the value of the input 
pixel 110. The output of the scaling function evaluator 120 is the intermediate 
value I. The intermediate value I is then passed to the tone scale applicator 124. 
The purpose of the tone scale applicator 124 is to affect the contrast of the value of 
the input pixel 110 with respect to the local region 112. The tone scale applicator 

20 124 modifies the intermediate value I, thereby creating the second intermediate 

value I2. The tone scale applicator 124 modifies the intermediate value I with the 
tone scale function 130 (i.e., f(x)). Thus, the operation of the tone scale applicator 
may be expressed in the following equation: I2 = f(I), where I2 is the second 
intermediate value, f(x) is the tone scale function, and I is the intermediate value 

25 output from the scaling function evaluator 120. 

The tone scale function f(x) is generated by the tone scale function 
generator 126. In the preferred embodiment, the tone scale function f(x) is the 
sigmoid function, formed by integrating a Gaussian probability distribution 
function, and is well known in the art. The sigmoid function is fully characterized 

30 by the standard deviation a of the Gaussian probability function that is integrated. 
The tone scale function f(x) may be expressed by the following formula: 
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In the preferred embodiment, the tone scale function is found by computing the 
summation of discrete samples of a Gaussian distribution, as shown in the 
following formula: 



2 

— X 



Z^exp 

fix) - 



u=a 



2a^ 



u=—a 



1 0 where 2a is the number of samples in the LUT. In the preferred embodiment, a = 
50. 

Notice that the slope of the tone scale function f(x) evaluated at x = 
0.5 may be determined by the relationship: 



1 

r = 



where y is the slope of the sigmoid function evaluated at x = 0.5, and the 
approximate slope of the summation described above, and a is the standard 
20 deviation of the Gaussian probability function. When the value of y> 1 .0, the 
operation sharpens the digital image channel by reshaping the local structure of 
edges. When the value of y is < 1.0, the operation tends to produce an image with 
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a softer appearance than the input image. In the preferred embodiment y = 2A and 
a = 0.19, In the preferred embodiment, the tone scale function f(x) is independent 
of the local statistics output from the local statistics determiner 116, and thus need 
be computed only one time per image channel, despite the number of pixels being 
5 processed. 

Figure 10 illustrates a variety of tone scale functions, one of which 
is shown as the tone scale function 130, and the associated standard deviations of 
the Gaussian probability function a. Those skilled in the art will recognize that a 
wide variety of functions in addition to the sigmoid function will accomplish the 

1 0 goal of the tone scale applicator 124. 

The second intermediate value h output from the tone scale 
applicator 124 is passed to the inverse scaling function applicator 128. In addition, 
the inverse scaling function s"^(x) output from the inverse scaling function 
generator 122 is passed to the inverse scaling function applicator 128. The 

1 5 purpose of the inverse scaling function applicator 128 is to generate the processed 
pixel value u' from the second intermediate value I2. The inverse scaling function 
applicator 1 28 generates the enhanced pixel value u* by evaluating the inverse 
scaling function: 

20 u* = s-^(l2) 

Substituting for I2 and I, 

u' = s-Vf{s(u)}) 

25 

where s'*(x) represents the inverse scaling function, f{x} represents the tone scale 
function, s(x) represents the scaling function, u represents the value of the input 
pixel 110, and u' represents the value of the enhanced pixel. The digital image 
channel output from the pedestal modified 60 is an enhanced pedestal signal 
30 n*ped(x,y). 
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Typically, the process for generating a processed pixel value u* from 
the pedestal modifier 60 is repeated for every pixel of the digital image channel in 
order to generate a processed pixel value corresponding to each pixel in the digital 
image channel. These processed pixel values, when considered as a collection, 
5 form a processed digital image channel. Altematively, the process for generating a 
processed pixel value u* may be repeated for a subset of every pixel of the digital 
image channel in order to generate a processed pixel value corresponding to a 
subset of pixels in the digital image channel. In the latter case, the subset, when 
considered as a collection, forms a processed image for a portion of the channel. In 

1 0 general, the parameters of the present algorithm may be adjusted in order to 

produce an enhanced digital image channel, with the appearance of greater edge 
contrast and more clarity than the digital image channel. 

The pedestal modifier 60 requires definition of two parameters: the 
size of the local region (which may be described by a window size for the 

1 5 sharpening filter) and the shape of the tone scale conversion function (which may 
be described by the slope of the tone scale fimction). The algorithm tends to 
perform best when the window size is kept fairly small, e.g., no greater than say 
about 7X7 pixels for a 1024 X 1536 imager. Indeed, as the window size and 
slope of the local tone scale fimction increase, the image begins to have a "paint- 

20 by-number" artifact. Also, a large window tends to compress texture on either side 
of a large occlusion boundary type edge in an image because the maximimi and the 
minimum tend to occur on opposite sides of the edge. This effect is reduced by 
keeping the window size relatively small. A window size of 5 X 5 or fewer pixels 
has been found generally preferable for producing satisfactory results in typical 

25 situations for a 1024 X 1536 resolution. As mentioned earlier, other suitable 

window sizes may be empirically determined without difficulty for other image 
resolutions; altematively, a larger window may be acceptable in certain situations 
for a given resolution if the user is willing to accept increased artifacts in order to 
obtain the sharpening effect of the larger window. 

30 The pedestal modifier as herein described has the advantage of 

controlling the value of the central pixel in accordance with a statistical 
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characteristic of the region, e.g., driving the value of the central pixel either toward 
the local maximum or the local minimum of the region, except in the case where 
the central pixel is substantially midway between the local maximum and the local 
minimum. Consequently, edge transitions occur over a narrower range of pixels 
than in the input image, thus generating an image that appears to be sharper, or 
more in focus, than the original image. Moreover, since the output of the tone 
scale conversion is modified by the statistical characteristic, e.g., bounded by the 
local maximum and the local minimum of the region, systematic overshoot and 
undershoot at an edge boundary is diminished and the ringing artifact is not as 
noticeable. 

A detailed view of the texture modifier 70 can be seen in Fig. 1 1 in 
accordance with the invention. The texture modifier 70 inputs the digital image 
channel n(x,y), the texture signal n^ifx^y), the luminance noise table 6 and the 
chrominance channels, gm(x,y) and ill(x,y). The output of the texture modifier 70 is 
an enhanced texture signal n'txt(x,y). The purpose of the texture modifier 70 is to 
boost the texture signal by a factor that is related to an estimate of the local signal 
to noise (SNR) ratio and an estimate of the local color, as determined by gm(x,y) 
and ill(x,y). 

Thus, disregarding local color, the portion of the texture signal 
coincident with regions of the digital image channel having high SNR will 
experience a greater level of boost as compared with the portion of the texture 
signal associated with low SNR regions of the digital image channel. For example, 
suppose that an image to be sharpened contains a large region of clear blue sky. 
The only modulation in this region is likely to be noise resulting fi-om the imaging 
system. Therefore, it is not desirable to increase to amplitude of this modulation. 
On the other hand, modulation of a grassy field is a result of the modulation due to 
many blades of grass in addition to image system noise. In this case, it is preferable 
to boost the modulation since it can be attributed to actual modulation in the scene. 
In the present invention, the signal to noise ratio is estimated to act as a classifier 
that may be used to distinguish between, e.g., the two aforementioned types of 
regions in a scene. 
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Considering only color, it may be desirable for the texture signal 



n 'txt(x,y) to be enhanced either more or less, depending on the local color about 
location (x,y). For example, any detail occurring on a clear blur sky may simply be 
from film grain. Thus, it would be desirable to deemphasize the sharpening that is 
5 applied to pixels in the original color digital image corresponding to the color blue. 
This may be accomplished by lowering the gain which is applied to the texture 
signal ntxt(x,y) for locations in the original color digital image with blue colors. 



i^txt(x,y) and the luminance noise table 6 are input to a local SNR estimator 160. 

1 0 The purpose of the local SNR estimator 160 is to estimate the local signal to noise 
ratio which will later be used to determine the level of boost given to a specific 
pixel of the texture signal. In the preferred embodiment, the output of the local 
SNR estimator 160 is one estimate of the SNR for each pixel in the digital image 
channel, represented by the symbol SNR(x,y). However, it is also possible that the 

1 5 SNR may be calculated for only a subset of the pixels' locations of the digital 
image channel. The local SNR estimator 160 will be described in greater detail 
hereinbelow. 



an SNR gain determiner 134. The SNR gain determiner 134 is meant to determine 
20 an individual level of boost B(x,y) to apply at each particular (x,y) location of the 
texture signal, given that an estimate of the local SNR is known. This is 
accomplished by the use of a look up table (LUT), for example the LUT shown as 
a plot in Fig. 15. In the preferred embodiment, this LUT is monotonically 
increasing. The LUT output (the gain) increases with the increase of the LUT input 
25 (the estimate of the SNR.) 



In this regard, the digital image channel n(x,y), the texture signal 



The output of the local SNR estimator 160, SNR(x,y), is passed into 



The equation for the LUT shown in Fig 15 is given as: 



B(x.y) 
B(x.y) 



Sfmin for SNR(x^y) <SNRn,in 

sfrnin+(sfn,ax-sfn,in)(5A«r^,j;;-SNR„,in)/(SNRn,ax-SNR 



30 



for SNRmin<5'7V/?fx,>;;<SNR, 
sfmax for SNR(x,y) > SNRn^x 



B(x.y) 
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Where sfmax is > sfmin, and SNRmax ^ SNRmin- In the preferred embodiment, sfmax is 
3.5, sfmin is 2.0, SNRmax is 2.0 and SmL^n is 0.4. 

Additionally, the chrominance channels gm(x,y) and ill(x,y) are 
5 input to the color gain determiner 135. The color gain determiner 135 calculates 
an individual level of boost Bc(x,y) to apply at each location (x,y) of the texture 
signal, dependent upon the local color as described by the chrominance channels 
gm(x,y) and ill(x,y) and the distance of location (x,y) from identified color region(s) 
in a chrominance color space. The local color can be calculated from the pixel of 
10 interest, e.g., the central pixel 110 shown in Figure 9, or it may be derived from the 
f n values of one or more pixels sampled from a local neighborhood of pixels 

p"; including the pixel of interest. In the preferred embodiment, the color gain 

determiner 135 finds the value of the boost Bc(x,y) by applying a 2 dimensional 
= LUT to the values gm(x,y) and ill(x,y). Typical values of Bc(x,y) range from 0 to 5. 

15 The LUT should be a smoothly varying ftinction of the 2 input variables. An 
M example of such fimctions is described in European Patent Specification 

p 0441558B1, granted January 7, 1998 and entitled "Improvements Relating to 

Control Data Arrays", which is incorporated herein by reference. Note that the 
color gain determiner 135 may altematively be a fimction of one or more variables 
20 related to color (rather than 2 as described in the preferred embodiment.) 

Typically, the color gain determiner 135 will be a function of 1, 2 or 3 variables, 
and may be applied as a multi-dimensional LUT. As an alternative embodiment, 
the color signals input to the color gain determiner 135 need not be equivaltnet to 
the chrominance channels output from the luminance/chrominance converter 10. 
25 For instance, the red, green, and blue color channels could be input to the color 
gain determiner 135 while the green-magenta and illuminant channels are output 
from the luminance/chrominance converter 10. Those skilled in the art will 
recognize that similar results may be achieved by applying a color transformation 
to the original color digital image preceding input of color channels into the color 
30 gain determined 135, and such a transformation does not significantly deviate from 
the description of the preferred embodiment. 
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in the preferred embodiment, the color gain determiner 135 appHes 
a two dimensional LUT to the chrominance channels gm(x,y) and ill(x,y). The 
multi dimensional LUT may be formed using multivariate Gaussian functions. 
Each multivariate Gaussian function will be referred to as a color region pole, and 
may be described by a mean and a covariance about the mean. Each color region 
pole represents a region in color space for which the texture sharpening 
characteristics are preferentially treated. A color region pole is represented by a 
mean vector m and a covariance matrix as are well known in the art of statistics. 
In the preferred embodiment, for color region pole 0, the mean vector mo is 



ill 



0 



The covariance matrix of a multidimensional function 



represented as /Wq = 
is well known in the art and for color region pole 0, = 



In 



the preferred embodiment, the mean vector m and the covariance matrix may be 
determined by a training procedure wherein an operator selects pixels of interest, 
having common material properties (for instance sky, skin, or grass) for one or 
more images. The mean and covariance of these selected pixels determine the 
mean m and covariance k of the color region pole. The actual numerical values of 
the mean vector m and the covariance matrix k are dependent upon the color and/or 
logarithmic transformations of the entire imaging chain. In the preferred 
embodiment, three color region poles are defined. Color region pole 0 represents 

[-261 ^ , [2000 400] ^ ^ . , . 
skin. In this case, = and k^ = . Color region pole 1 

° [-109] ^ [400 550 J 

[-501 [400 20 1 ^ , 

represents sky. In this case, /Wj = and = ^^^^ . Color region pole 

2 represents grass. In this case, = 

Thus the color gain determiner 135 determines a value Bc(xy) by 
evaluating the following expression: 
p 



'153' 




"800 


0 ■ 


and = 






-69 




0 


600 



25 5,(x,j;) = // + 5]//,.exp 



1=0 



-]^{w-mj k.{w-m.) 
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where P is the number of color region poles (in the preferred embodiment, P = 3), 
w is a vector representing the color of the local region of the image. In the 



a color distance. Note that the value of w may be computed using only the selected 
5 pixel undergoing processing. Alternatively, the value of w may be determined by 
using the selected pixel plus some number of surrounding pixels, for example by 
using an averaging procedure, as is well known in the art of image processing. 
Thus, the value of w may be determined either as a point operation or as a local 
neighborhood operation. Also, |a is a base value of the color gain. In the preferred 

1 0 embodiment, |i= 1 . Finally, |J.i is the preferred ratio of the sharpness of color region 
pole i. Color region poles (such as skin and sky) for which less sharpening is 
desired have values of i^i less than 0. Color region poles (such as grass) for which 
more sharpening is desired have values of more than 0. In the preferred 
embodiment, the values of |Xo , Mi , and |ii2 are -0.5, -0.7 and 0.65 respectively. 

1 5 Thus, the value of the color weighting factor or color gain factor, in the 
preferred embodiment ranges from approximately 0.3 to 1.65. 



Fig 15B shows an example of such a LUT. Note that the output value of this LUT 
can assume more than 2 values, and the LUT is said to be a near continuous 

20 function. Since the LUT is a near continuous function, the value of Be is a non- 
binary value. Applying a multidimensional LUT is well knovm in the art of image 
processing, and is described for example in US Patent No. 4,500,91 9, which 
issued February 19, 1985 and is entitled "Color Reproduction System", and which 
is incorporated herein by reference. In this display, white represents a value of 

25 approximately 1.65 and black represents a value of approximately 0.3. 



described as operating on chrominance channels corresponding to the original 
color digital image, it is possible that the chrominance channels may be modified 
with spatial filters or color transformations preceding the input of the chrominance 
30 channels to the color gain determiner 135. Moreover, although the color gain 



preferred embodiment, w = 



. Note that the operation (w-mi) represents 



Typically, the color gain determiner 135 is implemented as a LUT. 



Note that although the color gain determiner 135 has herein been 



determiner 135 has herein been described as calculating an individual level of 
boost Bc(x,y) *o ^PPly each location ^x,y) of the texture signal, dependent upon 
the local color as described by the chrominance channels gm(x,y) ill(x,y), the 
individual level of boost Bc(x,y) "^^Y instead, or in addition, be applied at each 
5 location (x,y) of the pedestal signal in order to further allow for independent 
control of enhancement applied to edges based on local color. 

The SNR gain factor B(x,y) output by the SNR gain determiner 1 34, 
the color gain factor Bc(x,y) output from the color gain determiner 135, and the 
texture signal ntxi(x,y) then passed to a texture booster 136. The purpose of the 
-| 0 texture booster 1 36 is to multiply each value of the texture signal ntxt(x,y) its 
associated gain. The output of the texture booster 136 is the enhanced texture 
signal n 'txt(x.yh ^^^^ equation: 

n'^^(x,y) = n^^(x,y) * [iB(x^y)'Bj*B,(x^y)^Bm ] 

-| 5 where is a coefficient representing a minimum boost factor, which is 1 in the 
preferred embodiment. The effect, consequently, is that the color weighting or 
gain factor b^ causes an amplification (or a reduction) of high spatial frequency 
information for pixels that are closer in color to certain one(s) of the identified 
color regions, and a reduction of high spatial frequency information for pixels that 

20 ^e closer in color to certain other one(s) of the identified color regions. In other 
words, the color weighting factor Bc(x,y) causes more (or less) noise reduction for 
pixels of interest that are closer to certain one(s) of the identified color regions. 

Note that those skilled in the art will recognize that there are many 
equations by which a color gain factor Bc(x,y) ^i^Y be used with a SNR gain factor 

25 B(x,y) to enhance a texture signal (or signal comprising mostly of detail.) This 

enhanced texture signal n*txt(x,y) output from the texture modifier 70 is then passed 
to the adder 80 in Fig. 3 (or to the avoidance adder 81 in the case of the alternative 
embodiment shown in Fig. 4.) Note when {B(x,y)-Bjr)^Bc(x,y)'^Bm ^ the 
magnitude of the enhanced signal is less than the magnitude of the original texture 

30 signal \ntjct(x,y)\'^\n'ixt(x,y)\' Thus, if the present invention is tuned such that in 
noisy regions, iB(x,y)-Bm^*Bc(x,y)'^Bm ^^ O, the processed digital image chaimel 
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will appear to have less noise than the digital image channel input to the preference 
sharpener 2. 

A detailed block diagram of the local SNR estimator 160 may be 
seen in Fig 12. The digital image channel n(x,y) is first input to a local variance 
5 computer 170. The purpose of the local variance computer 170 is to attain a local 
estimate of activity. In the preferred embodiment, the local variance computer 1 70 
calculates the local variance cjr^(jc,j^) over a windov^ of size 7 by 7 pixels, although 
it has been determined that alternately sized windows perform the task with nearly 
equivalent results. 

1 0 The computation of the local variance of a set of values is well 

known in the art, and may be expressed as: 

OK^x^y) = Zpix+i.yyf /49 - (Zp(x+/,j.+/')/49)' 

1 5 Where / and j both range fi-om -3 to +3, inclusive, for a total of 49 pixels included 
in the computation of the variance aR^(x,y). Calculation of the variance at edges 
involves mirroring, a technique well known in the art of image processing. The 
output of the local variance computer 1 70 is a local variance for each pixel in the 
digital image channel. 

20 A subtracter 172 inputs both the digital image channel n(x,y) and 

the texture signal ntxt(x,y). The output of the subtracter 172 is the pedestal signal 
nped(x,y). The subtracter 172 determines the pedestal signal by subtracting the 
texture signal ntxt(x,y) fi-om the digital image channel n(x,y), according to the 
formula: 

25 nped(x,y) = n(x,y) - ntxt(x,y) 

Note that this pedestal signal nped(x,y) is identical to the pedestal signal determined 

by the pedestal splitter 50. In a practical implementation of the present invention, 

the pedestal signal need only be computed once. 

The noise table 6 and the pedestal signal nped(x,y) are input an 
30 expected noise determiner 190. The noise table contains the relationship between 

expected noise and intensity / of the digital image channel n(x,y). In the 
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preferred embodiment, the noise table is a two column table, a sample of which is 
shown hereinbelow in a two column list. The first column represents intensity 
level and the second column represents the expected standard deviation of noise 
for that intensity level. 
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The noise table may also be specified in terms of a four column 
table, where the first column is the intensity i, and the second to fourth columns 
correspond to the standard deviations of noise expected in the red, green, and blue 
channels, respectively, of the digital image channel input to the preference 
sharpener 2. In this case, the expected standard deviation of noise for intensity 
level i of the luminance channel n(x,y) is dependent upon the coefficients of the 
color rotation matrix used to generate the luminance channel, as given by: 



an(i) = sqrt((ar(i)*rcoef)^ + (ag(i)*gcoef) ^ + (ab(i)*bcoef) ^ )) 



where an(i) is the expected standard deviation of noise at intensity i for the 
luminance digital image channel n(x,y). Likewise, ar(i), ag(i), ab(i) are the 
expected standard deviations of noise of the respective red, green, and blue digital 
image channels at intensity i, and rcoef, gcoef, and bcoef are the coefficients of the 
1 5 color rotation matrix used to generate the luminance digital image channel n(x,y). 
As previously mentioned, in the preferred embodiment all three of these 
coefficients equal 1/3. Consequently, the luminance channel given in the preferred 
embodiment is given by: 

20 an(i) - l/3*sqrt(a,(i)' + ag(i) ' + ab(i) ' ) 

Thus a noise table corresponding to the luminance digital image channel n(x,y) 
may be either supplied or calculated by the expected noise determiner 190. 

The expected noise determiner 190 outputs an intensity dependent 
25 estimate of the standard deviation of noise oi^(x,y) for the intensity level of the 

pedestal signal nped(x,y). The expected noise determiner 190 either performs a table 
look-up, which relates the intensity of the pedestal signal to the expected standard 
deviation of noise o^(x,y)^ or calculates the expected noise, as described above. 
The output <5\<i(x,y) of the expected noise determiner 190 may be represented as: 

30 



C5i^(xy) = C5^{nped(x,y)), 
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If the value of nped(x,y) is not coincident with a value of / for which OrSf) exists, 
then linear interpolation is used to determine a value for Oy^{nped(x,y)). Linear 
interpolation is well known in the art and will not be further discussed. A plot of a 
5 noise table can be seen in Fig. 14. 

The procedure for generating a noise table such as the one 
illustrated in Fig. 14 can be found in U.S. Patent No, 5,641,596 issued June 24, 
1997 to Gray et al. or in U.S. Patent No. 5,923,775 issued July 13, 1999 to Snyder 
et al., both of which are incorporated herein by reference. 

1 0 An SNR calculator 1 80 inputs both the expected standard deviation 

of noise <5n(x,y) output from the expected noise determiner 190, and the local 
variance a^ix^y) output from the local variance computer 170. The SNR 
calculator 180 estimates a local signal to noise ratio SNR(x,j^) by computing the 
ratio of the signal variance (assuming that the local variance i3Y^{x,y) is the sum {in 

15 an RMS sense} of the variance due to signal and the variance due to noise.) Thus, 
the output of the SNR calculator 180 is an estimate of the local SNR at each pixel 
location, according to the following formula: 



20 



SNR(^,>;) = sqrt( aR {x,y) - aw {x,y) ) / on(x,y). 



Note that before attempting the square root operation, the value of csk {x,y) - 
<yn^(x,y) is checked to ensure that it is positive. If it is not, the value of SNR {x,y) 
is set to equal 0.0. 

Notice that many other estimates of the signal to noise ratio may be 

25 formulated. For example, an altemative embodiment of the local SNR estimator 

160 is shown in Fig. 13. The purpose of this embodiment is to provide an estimate 
of the local SNR by a means that is computationally less expensive that by that 
shovra in Fig. 12. In this altemative embodiment, the noise and signal deviation 
are both assumed to be Gaussian distributed, which allows for the use of mean 

30 absolute deviation (MAD) rather than standard deviation. The calculation of mean 
absolute deviation does not involve the computationally expensive square root and 
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squaring operations. In this embodiment, the texture signal ntjct(x,y) and the digital 
image channel n(x,y) are again input to the subtractor 172 for the purpose of 
producing the pedestal signal nped(x,y), as previously described. The pedestal 
signal nped(x,y) will again be used as an input to the expected noise determiner 190 
5 to determine the value ofoi^(x,y). 

The texture signal ntxt(x,y) is input to the local MAD calculator 200 
in order to determine the local mean absolute deviation in the texture signal. By 
assuming that the mean value of the texture signal ntxt(x,y) is 0, the formula for 
computing the local MAD used by the local MAD calculator 200 is given as: 

10 

MADR(x,y) = Z| nUx+i .y^J) I /49 

where i and j preferably both range from -3 to +3, for a total of 49 pixels included 
in the calculation of the mean absolute deviation. The local MAD calculator 200 

1 5 outputs the value of the mean absolute deviation MADR(x,y), which will later be 
used in calculations necessary for determining the estimate of the local SNR. 

The purpose of a noise table adjuster 210 is to compensate for the 
differences between calculating MADR(x,y) of the texture signal, and calculating 
the standard deviation <3YL(x,y) of the digital image channel. The noise table 

20 adjuster 210 modifies the luminance noise table for the purpose of using mean 
absolute deviation, rather than standard deviation. Assuming, as previously 
mentioned, that the limiinance noise table contains the relationship between 
expected noise an(%> and intensity U It is known in the art that the MAD of a 
Gaussian distribution is related to the standard deviation a of that distribution by 

25 the following relationship: 

MAD =sqrt(2/7r)a 

Thus, in order to translate the luminance noise table from the metric of standard 
30 deviation to the metric of MAD, each value of crn(i) must be scaled by about 0.8, 
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In addition, there is the difference between this alternative 
embodiment and the preferred embodiment concerning which signal is used for 
computation of the local activity measure. In the preferred embodiment, the 
variance of the digital image channel is computed. In this alternative embodiment, 
5 the texture signal is used to calculate the value of MADR(x,y), providing a 
computational benefit when the signal mean is assumed to be zero. Thus the 
relationship of the texture signal ntxt(x,y) to the digital image channel n(x,y) must 
be understood in order to adjust the luminance noise table to account for this. This 
relationship can be easily expressed only if the digital filtering process used to 

1 0 generate the texture signal is linear. For the sake of this step, the effects of the 
artifact avoidance signal are ignored, and the texture signal is assumed to be 
equivalent to the highpass signal. For a general (n x m) filter with coefficients 
h(ij), wherein i = -(«-l)/2, -(w-3)/2,...,«-l/2 and J = -(m-l)/2, -(/w-3)/2,...,m-l/2, the 
variance of the filtered signal is related to the variance of the original signal by the 

1 5 following relationship : 

where cr^fs represents the variance of the filtered signal, a^os represents the 
20 variance of the original signal, and the summation occurs over / and j\ 

Thus, in the case of the present invention, the filter h(ij) is assumed 
to be expressed as: 



25 



hOJ) =S(ij)~g(iJ) 



where is the delta function which has a value of one at / = 0 and 7 = 0 and a 
value of zero everywhere else. This expression accounts for the previously 
described process of generating the texture signal (ignoring the artifact avoidance 
signal) by blurring the digital image channel with a Gaussian filter g-fzj^, then 
30 subtracting the result fi-om the digital image channel. Thus, considering this 

description, the noise table adjuster 210 must scale each value of the luminance 
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noise table by a factor to account for the estimation of local activity on the texture 
signal rather than the digital image channel. This factor / can be expressed as: 

f=scin{TLh(ij)') 

5 

where the summation occurs over / and y, and h(ij) is as previously described. The 
square root of the result is taken because the luminance noise table is in the units 
of standard deviation, rather than variance. 

Thus, the operation of the noise table adjuster 210 is to scale each 
1 0 entry an(0 of the luminance noise table by a factor m equal to 

m =/* sqrt ( lln) 

in order to compensate for the calculation of MAD rather than standard deviation, 
1 5 and the calculation with the texture signal rather then the digital image channel. 

The modified noise table output from the noise table adjuster 210 is 
passed as an input to the expected noise determiner 190 for the purpose of 
attaining an estimate of the expected MAD from noise, MADN(x,y). The expected 
noise determiner 190 uses the modified noise table to determine the value of 
20 MADN(x,y) for each value of npQ^(x,y), The output of the expected noise 

determiner 190 is the value of MADN(x,y) at each location of the digital image 
chaimel. 

The SNR calculator 180 of the alternative embodiment shown in 
Fig. 13 acts essentially the same as the SNR calculator 180 of the preferred 
25 embodiment. The purpose of the SNR calculator 180 is to output a value of 

estimated SNR at each location of the digital image channel. To this end, the SNR 
calculator 1 80 calculates the following formula: 

SNR(x,y) = sqrt(MADR(x,y)^ - MADN(x,y)^ ) / MADN(x,y) 

30 




-41- 



The estimate of the local SNR is output from the local SNR estimator 160 and 
input to the gain detemiiner 134, shown in Fig. 1 1 in order to allow for the 
determination of the appropriate boost factor for the texture signal at each location. 

In summary, the use of the pedestal splitter 50, decomposing the 
5 image into a signal containing the image detail and another signal primarily 
containing the large image edges, allows for independent control of the 
enhancement applied to edges and texture. Edges are reshaped by a morphological 
operation in order to increase edge contrast and avoid boundary artifacts. Texture 
in enhanced with regard to an estimate of the local signal to noise ratio and, in 

1 0 accordance with the invention, with regard to an estimate of local color. 

The present invention has been described with reference to a 
preferred embodiment. Changes may be made to the preferred embodiment 
without deviating from the scope of the present invention. For instance, in the 
preferred embodiment, a calculation of the local SNR is made for each pixel of the 

1 5 digital image channel. However, it is conceivable that the calculation of the SNR 
could be made for only every N*^ pixel in order to reduce computational cost, and 
replicated or interpolated. Calculation of the local SNR was described with 
reference to a local variance calculation on the digital image channel n(x,y). 
Additionally, an alternative embodiment described a method by which local SNR 

20 may be estimated based upon using a MAD calculation upon the textxire signal 
^txt(x,y)- Those skilled in the art will recognize that the local SNR may also be 
estimated by performing a variance calculation on the texture signal, or by 
performing a MAD calculation upon the digital image channel n(x,y). Such 
modifications to the preferred embodiment do not significantly deviate from the 

25 scope of the present invention. 
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PARTS LIST 
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scaling function evaluator 
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